Skip to content

⚡️ Speed up function discover_parameters_unittest by 252% in PR #753 (test_cache_revival) #754

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Parent: Test cache revival
wants to merge 4 commits into from
Closed

⚡️ Speed up function discover_parameters_unittest by 252% in PR #753 (test_cache_revival) #754

wants to merge 4 commits into from
+28 −32

Conversation

@codeflash-ai
Copy link
Contributor

@codeflash-ai codeflash-ai bot commented Sep 23, 2025

⚡️ This pull request contains optimizations for PR #753

If you approve this dependent PR, these changes will be merged into the original PR branch test_cache_revival.

This PR will be automatically closed if the original PR is merged.

📄 252% (2.52x) speedup for discover_parameters_unittest in codeflash/discovery/discover_unit_tests.py

⏱️ Runtime : 350 microseconds → 99.4 microseconds (best of 264 runs)

⚡️ This change will improve the performance of the following benchmarks:

{benchmark_info_improved}

🔻 This change will degrade the performance of the following benchmarks:

Benchmark File :: Function Original Runtime Expected New Runtime Slowdown
tests.benchmarks.test_benchmark_merge_test_results::test_benchmark_merge_test_results 44.2 milliseconds 44.3 milliseconds 0.12%

📝 Explanation and details

The optimized code achieves a 252% speedup through two key optimizations:

1. Early Exit for No Underscores: Added a check if '_' not in function_name: that immediately returns False for strings without underscores. This avoids unnecessary splitting operations for simple function names, providing significant speedups (51-69% faster) for cases like single words or empty strings.

2. Right-Split Optimization: Replaced split("_") with rsplit("_", 1) which only splits from the right once, creating exactly 2 parts instead of potentially hundreds. This dramatically reduces memory allocation and processing time, especially for long function names with many underscores.

Performance Impact by Test Type:

  • Simple cases (no underscores): 51-69% faster due to early exit
  • Valid numeric suffixes: 21-50% faster from efficient rsplit
  • Large-scale tests: 660-3584% faster - the rsplit optimization shines here, avoiding expensive operations on strings with hundreds of parts
  • Edge cases with non-numeric suffixes: Slight 2-13% slowdown due to the additional underscore check, but this is minimal compared to the gains

The optimizations preserve exact functionality while being most effective for complex function names with many parts, which are common in parameterized test scenarios.

✅ Correctness verification report:

Test Status
⚙️ Existing Unit Tests 🔘 None Found
🌀 Generated Regression Tests ✅ 61 Passed
⏪ Replay Tests ✅ 100 Passed
🔎 Concolic Coverage Tests 🔘 None Found
📊 Tests Coverage 100.0%
🌀 Generated Regression Tests and Runtime 
from __future__ import annotations

# imports
import pytest  # used for our unit tests
from codeflash.discovery.discover_unit_tests import \
    discover_parameters_unittest

# unit tests

# --------------------------
# Basic Test Cases
# --------------------------

def test_basic_valid_with_numeric_suffix():
    # Should detect numeric suffix and split correctly
    codeflash_output = discover_parameters_unittest("foo_123"); result = codeflash_output # 1.29μs -> 992ns (30.2% faster)

def test_basic_valid_with_long_name_and_numeric_suffix():
    # Should handle longer names
    codeflash_output = discover_parameters_unittest("bar_baz_qux_456"); result = codeflash_output # 1.44μs -> 962ns (50.0% faster)

def test_basic_no_numeric_suffix():
    # Should return False if no numeric suffix
    codeflash_output = discover_parameters_unittest("foo_bar"); result = codeflash_output # 852ns -> 972ns (12.3% slower)

def test_basic_single_word():
    # Should return False for single word with no underscore
    codeflash_output = discover_parameters_unittest("foobar"); result = codeflash_output # 621ns -> 411ns (51.1% faster)

def test_basic_numeric_suffix_not_last():
    # Should not match if numeric is not last part
    codeflash_output = discover_parameters_unittest("foo_123_bar"); result = codeflash_output # 902ns -> 1.03μs (12.6% slower)

# --------------------------
# Edge Test Cases
# --------------------------

def test_edge_empty_string():
    # Should handle empty string
    codeflash_output = discover_parameters_unittest(""); result = codeflash_output # 611ns -> 371ns (64.7% faster)

def test_edge_underscore_only():
    # Should handle string of underscores
    codeflash_output = discover_parameters_unittest("_"); result = codeflash_output # 832ns -> 892ns (6.73% slower)

def test_edge_numeric_only():
    # Should handle string that is only numeric
    codeflash_output = discover_parameters_unittest("123"); result = codeflash_output # 602ns -> 391ns (54.0% faster)

def test_edge_underscore_and_numeric():
    # Should handle underscore and numeric only
    codeflash_output = discover_parameters_unittest("_123"); result = codeflash_output # 1.31μs -> 1.02μs (28.6% faster)

def test_edge_multiple_underscores_and_numeric():
    # Should handle multiple underscores before numeric
    codeflash_output = discover_parameters_unittest("___123"); result = codeflash_output # 1.44μs -> 952ns (51.6% faster)

def test_edge_trailing_underscore_no_numeric():
    # Should not match if trailing underscore with no numeric
    codeflash_output = discover_parameters_unittest("foo_"); result = codeflash_output # 841ns -> 972ns (13.5% slower)

def test_edge_leading_underscore_and_numeric_suffix():
    # Should handle leading underscore and valid numeric suffix
    codeflash_output = discover_parameters_unittest("_foo_123"); result = codeflash_output # 1.47μs -> 981ns (50.1% faster)

def test_edge_numeric_suffix_with_leading_zeros():
    # Should handle numeric suffix with leading zeros
    codeflash_output = discover_parameters_unittest("foo_00123"); result = codeflash_output # 1.26μs -> 1.01μs (24.8% faster)

def test_edge_non_numeric_suffix():
    # Should not match if last part is not purely digits
    codeflash_output = discover_parameters_unittest("foo_12a"); result = codeflash_output # 912ns -> 1.03μs (11.6% slower)

def test_edge_suffix_is_negative_number():
    # Should not match if last part is negative number (not purely digits)
    codeflash_output = discover_parameters_unittest("foo_-123"); result = codeflash_output # 911ns -> 1.03μs (11.7% slower)

def test_edge_suffix_is_float():
    # Should not match if last part is a float
    codeflash_output = discover_parameters_unittest("foo_123.45"); result = codeflash_output # 971ns -> 1.03μs (5.82% slower)

def test_edge_suffix_is_alphanumeric():
    # Should not match if last part is alphanumeric
    codeflash_output = discover_parameters_unittest("foo_123abc"); result = codeflash_output # 941ns -> 1.00μs (6.09% slower)

def test_edge_suffix_is_empty_after_underscore():
    # Should not match if there's an empty last part
    codeflash_output = discover_parameters_unittest("foo_"); result = codeflash_output # 831ns -> 921ns (9.77% slower)

def test_edge_multiple_numeric_suffixes():
    # Should only consider the last part as the numeric suffix
    codeflash_output = discover_parameters_unittest("foo_123_456"); result = codeflash_output # 1.47μs -> 1.01μs (45.6% faster)

def test_edge_all_underscores():
    # Should handle string with only underscores
    codeflash_output = discover_parameters_unittest("____"); result = codeflash_output # 882ns -> 901ns (2.11% slower)

def test_edge_underscore_and_non_numeric():
    # Should not match if last part is non-numeric after underscore
    codeflash_output = discover_parameters_unittest("_foo_bar"); result = codeflash_output # 972ns -> 992ns (2.02% slower)

# --------------------------
# Large Scale Test Cases
# --------------------------

def test_large_scale_many_parts_with_numeric_suffix():
    # Should handle long function names with many underscores and numeric suffix
    name = "_".join(["part"] * 999) + "_42"
    codeflash_output = discover_parameters_unittest(name); result = codeflash_output # 49.8μs -> 1.35μs (3584% faster)

def test_large_scale_many_parts_without_numeric_suffix():
    # Should handle long function names with many underscores but no numeric suffix
    name = "_".join(["part"] * 1000)
    codeflash_output = discover_parameters_unittest(name); result = codeflash_output # 27.8μs -> 1.51μs (1735% faster)

def test_large_scale_numeric_suffix_is_large_number():
    # Should handle very large numeric suffix
    name = "foo_" + "9" * 100
    codeflash_output = discover_parameters_unittest(name); result = codeflash_output # 1.51μs -> 1.24μs (21.7% faster)

def test_large_scale_all_numeric_parts():
    # Should only match last part as numeric suffix
    name = "_".join(["123"] * 999) + "_456"
    codeflash_output = discover_parameters_unittest(name); result = codeflash_output # 38.1μs -> 1.29μs (2844% faster)

def test_large_scale_all_numeric_parts_no_suffix():
    # Should not match if last part is not numeric
    name = "_".join(["123"] * 999) + "_foo"
    codeflash_output = discover_parameters_unittest(name); result = codeflash_output # 27.4μs -> 1.37μs (1899% faster)

def test_large_scale_empty_parts():
    # Should handle many empty parts before numeric suffix
    name = ("_" * 999) + "_789"
    codeflash_output = discover_parameters_unittest(name); result = codeflash_output # 20.3μs -> 1.32μs (1438% faster)
    # The split will result in a lot of empty strings, last part is '789'
    expected_prefix = "_" * 998  # 999 underscores split gives 1000 parts, last is '789'

def test_large_scale_suffix_is_zero():
    # Should handle suffix '0'
    name = "foo_" + "0"
    codeflash_output = discover_parameters_unittest(name); result = codeflash_output # 1.25μs -> 1.05μs (19.0% faster)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.
#------------------------------------------------
from __future__ import annotations

# imports
import pytest  # used for our unit tests
from codeflash.discovery.discover_unit_tests import \
    discover_parameters_unittest

# unit tests

# --- Basic Test Cases ---

def test_basic_with_numeric_suffix():
    # Function name with numeric suffix
    codeflash_output = discover_parameters_unittest("process_data_42"); result = codeflash_output # 1.65μs -> 1.12μs (47.2% faster)

def test_basic_without_numeric_suffix():
    # Function name without numeric suffix
    codeflash_output = discover_parameters_unittest("process_data"); result = codeflash_output # 982ns -> 1.01μs (2.96% slower)

def test_basic_single_word():
    # Single word function name, no underscores
    codeflash_output = discover_parameters_unittest("data"); result = codeflash_output # 631ns -> 380ns (66.1% faster)

def test_basic_multiple_underscores_and_numeric_suffix():
    # Multiple underscores with numeric suffix
    codeflash_output = discover_parameters_unittest("load_test_case_100"); result = codeflash_output # 1.56μs -> 1.14μs (36.9% faster)

def test_basic_numeric_suffix_is_zero():
    # Numeric suffix is zero
    codeflash_output = discover_parameters_unittest("run_simulation_0"); result = codeflash_output # 1.54μs -> 1.08μs (42.5% faster)

# --- Edge Test Cases ---

def test_edge_empty_string():
    # Empty string as function name
    codeflash_output = discover_parameters_unittest(""); result = codeflash_output # 630ns -> 371ns (69.8% faster)

def test_edge_underscore_only():
    # Function name is just underscores
    codeflash_output = discover_parameters_unittest("_"); result = codeflash_output # 882ns -> 922ns (4.34% slower)

def test_edge_trailing_underscore():
    # Function name ends with underscore
    codeflash_output = discover_parameters_unittest("process_data_"); result = codeflash_output # 862ns -> 992ns (13.1% slower)

def test_edge_leading_underscore():
    # Function name starts with underscore
    codeflash_output = discover_parameters_unittest("_42"); result = codeflash_output # 1.26μs -> 1.00μs (25.9% faster)

def test_edge_multiple_numeric_parts():
    # Function name with multiple numeric parts, only last is considered
    codeflash_output = discover_parameters_unittest("test_1_2_3"); result = codeflash_output # 1.55μs -> 1.01μs (53.5% faster)

def test_edge_suffix_is_not_integer():
    # Suffix is not an integer
    codeflash_output = discover_parameters_unittest("parse_data_final"); result = codeflash_output # 972ns -> 1.08μs (10.2% slower)

def test_edge_suffix_is_float():
    # Suffix is a float
    codeflash_output = discover_parameters_unittest("parse_data_3.14"); result = codeflash_output # 952ns -> 1.07μs (11.2% slower)

def test_edge_suffix_is_negative_integer():
    # Suffix is negative integer (not handled by isdigit)
    codeflash_output = discover_parameters_unittest("parse_data_-42"); result = codeflash_output # 902ns -> 962ns (6.24% slower)

def test_edge_suffix_is_alphanumeric():
    # Suffix is alphanumeric
    codeflash_output = discover_parameters_unittest("parse_data_42a"); result = codeflash_output # 911ns -> 1.01μs (9.98% slower)

def test_edge_suffix_is_empty_after_split():
    # Function name ends with underscore, so last part is empty string
    codeflash_output = discover_parameters_unittest("function_"); result = codeflash_output # 841ns -> 922ns (8.79% slower)

def test_edge_suffix_is_large_number():
    # Suffix is a very large number
    codeflash_output = discover_parameters_unittest("big_data_999999999"); result = codeflash_output # 1.64μs -> 1.09μs (50.5% faster)

def test_edge_suffix_is_zero_padded():
    # Suffix is zero-padded
    codeflash_output = discover_parameters_unittest("test_case_0001"); result = codeflash_output # 1.46μs -> 982ns (49.0% faster)

# --- Large Scale Test Cases ---

def test_large_scale_long_function_name_with_numeric_suffix():
    # Very long function name with numeric suffix at the end
    long_name = "a_" * 499 + "b_123"
    expected_prefix = "a_" * 499 + "b"
    codeflash_output = discover_parameters_unittest(long_name); result = codeflash_output # 12.2μs -> 1.29μs (841% faster)

def test_large_scale_long_function_name_without_numeric_suffix():
    # Very long function name without numeric suffix
    long_name = "x_" * 999 + "y"
    codeflash_output = discover_parameters_unittest(long_name); result = codeflash_output # 10.5μs -> 1.38μs (660% faster)

def test_large_scale_many_underscores_and_numeric_suffix():
    # Function name with many underscores and numeric suffix
    parts = ["part{}".format(i) for i in range(999)]
    function_name = "_".join(parts) + "_77"
    expected_prefix = "_".join(parts)
    codeflash_output = discover_parameters_unittest(function_name); result = codeflash_output # 60.7μs -> 1.66μs (3548% faster)

def test_large_scale_suffix_is_max_length_integer():
    # Suffix is a very large integer string (close to 1000 digits)
    large_num = "9" * 999
    function_name = "data_" + large_num
    codeflash_output = discover_parameters_unittest(function_name); result = codeflash_output # 2.93μs -> 2.60μs (12.8% faster)

def test_large_scale_numeric_suffix_in_middle():
    # Numeric part in the middle, not at end
    function_name = "part_123_part"
    codeflash_output = discover_parameters_unittest(function_name); result = codeflash_output # 982ns -> 1.00μs (2.00% slower)

def test_large_scale_all_numeric_parts():
    # All parts are numeric except the first
    function_name = "a_1_2_3_4_5_6_7_8_9_10"
    expected_prefix = "a_1_2_3_4_5_6_7_8_9"
    codeflash_output = discover_parameters_unittest(function_name); result = codeflash_output # 1.60μs -> 992ns (61.6% faster)

def test_large_scale_suffix_is_empty_string():
    # Suffix is an empty string after split
    function_name = "x_" * 999 + ""
    codeflash_output = discover_parameters_unittest(function_name); result = codeflash_output # 10.5μs -> 1.37μs (662% faster)

# --- Additional Edge Cases ---

def test_edge_suffix_is_space():
    # Suffix is a space
    codeflash_output = discover_parameters_unittest("test_case_ "); result = codeflash_output # 891ns -> 972ns (8.33% slower)

def test_edge_suffix_is_tab():
    # Suffix is a tab character
    codeflash_output = discover_parameters_unittest("test_case_\t"); result = codeflash_output # 811ns -> 922ns (12.0% slower)

def test_edge_suffix_is_special_characters():
    # Suffix is special characters
    codeflash_output = discover_parameters_unittest("test_case_!@#"); result = codeflash_output # 962ns -> 991ns (2.93% slower)

def test_edge_suffix_is_unicode_digits():
    # Suffix is unicode digits (e.g., Arabic-Indic digits)
    codeflash_output = discover_parameters_unittest("test_case_١٢٣"); result = codeflash_output # 2.00μs -> 1.56μs (28.2% faster)

def test_edge_suffix_is_mixed_unicode_and_ascii_digits():
    # Suffix is mixed unicode and ascii digits
    codeflash_output = discover_parameters_unittest("test_case_1٢3"); result = codeflash_output # 1.71μs -> 1.41μs (21.3% faster)

def test_edge_suffix_is_newline():
    # Suffix is a newline character
    codeflash_output = discover_parameters_unittest("test_case_\n"); result = codeflash_output # 851ns -> 902ns (5.65% slower)

def test_edge_suffix_is_multiple_underscores():
    # Suffix is multiple underscores
    codeflash_output = discover_parameters_unittest("test_case___"); result = codeflash_output # 912ns -> 952ns (4.20% slower)

# --- Determinism Test ---

def test_determinism():
    # Multiple calls with same input should yield same output
    input_name = "repeatable_123"
    codeflash_output = discover_parameters_unittest(input_name); result1 = codeflash_output # 1.29μs -> 1.00μs (28.9% faster)
    codeflash_output = discover_parameters_unittest(input_name); result2 = codeflash_output # 591ns -> 621ns (4.83% slower)
# codeflash_output is used to check that the output of the original code is the same as that of the optimized code.
⏪ Replay Tests and Runtime
Test File::Test Function Original ⏱️ Optimized ⏱️ Speedup
benchmarks/codeflash_replay_tests_cv_23o2q/test_tests_benchmarks_test_benchmark_merge_test_results__replay_test_0.py::test_codeflash_discovery_discover_unit_tests_discover_parameters_unittest_test_benchmark_merge_test_results 33.9μs 35.6μs -4.67%⚠️

To edit these changes git checkout codeflash/optimize-pr753-2025-09-23T14.02.28 and push.

Codeflash

KRRT7 and others added 4 commits September 23, 2025 06:31
@KRRT7
oops
e784545
@KRRT7
respond to code review
b0da369
@KRRT7
cleanup
ce72cfd
@codeflash-ai
Optimize discover_parameters_unittest
fda19ea 
The optimized code achieves a 252% speedup through two key optimizations:

**1. Early Exit for No Underscores**: Added a check `if '_' not in function_name:` that immediately returns `False` for strings without underscores. This avoids unnecessary splitting operations for simple function names, providing significant speedups (51-69% faster) for cases like single words or empty strings.

**2. Right-Split Optimization**: Replaced `split("_")` with `rsplit("_", 1)` which only splits from the right once, creating exactly 2 parts instead of potentially hundreds. This dramatically reduces memory allocation and processing time, especially for long function names with many underscores.

**Performance Impact by Test Type**:
- **Simple cases** (no underscores): 51-69% faster due to early exit
- **Valid numeric suffixes**: 21-50% faster from efficient rsplit
- **Large-scale tests**: 660-3584% faster - the rsplit optimization shines here, avoiding expensive operations on strings with hundreds of parts
- **Edge cases with non-numeric suffixes**: Slight 2-13% slowdown due to the additional underscore check, but this is minimal compared to the gains

The optimizations preserve exact functionality while being most effective for complex function names with many parts, which are common in parameterized test scenarios.
@codeflash-ai codeflash-ai bot added the ⚡️ codeflash Optimization PR opened by Codeflash AI label Sep 23, 2025
@codeflash-ai codeflash-ai bot mentioned this pull request Sep 23, 2025
Merged
@KRRT7 KRRT7 force-pushed the test_cache_revival branch from ce72cfd to bbc630f Compare September 23, 2025 14:06
@KRRT7 KRRT7 closed this Sep 23, 2025
@codeflash-ai codeflash-ai bot deleted the codeflash/optimize-pr753-2025-09-23T14.02.28 branch September 23, 2025 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

⚡️ codeflash Optimization PR opened by Codeflash AI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@KRRT7